End-to-end distributed tracing with external telemetry data

ABSTRACT

Techniques are described for generating an end-to-end distributed trace in connection with a cloud or datacenter environment. In one example, a server obtains target application telemetry data and external telemetry data associated with one or more correlation identifiers included in one or more network communications provided to a target application in the cloud or datacenter environment. The server aggregates the target application telemetry data and the external telemetry data based on the one or more correlation identifiers to generate an end-to-end distributed trace associated with the one or more network communications.

TECHNICAL FIELD

The present disclosure relates to computer networking.

BACKGROUND

Today, application owners commonly arrange for cloud or datacenteroperators to run applications in cloud or datacenter environments. Thispermits the application owners to devote resources to maintaining andimproving the cloud- or datacenter-native applications themselves, whilethe cloud or datacenter operators manage the environments in which theapplications are running. End users can access applications hosted incloud or datacenter environments via the Internet.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system configured for end-to-end distributedtracing, according to an example embodiment.

FIG. 2 illustrates a graphical depiction of how components of the systemof FIG. 1 can contribute to an end-to-end distributed trace, accordingto an example embodiment.

FIG. 3 illustrates a system configured for end-to-end distributedtracing using techniques specified by the OpenTelemetry™ tracingspecification, according to an example embodiment.

FIG. 4 illustrates a display of an end-to-end distributed trace,according to an example embodiment.

FIG. 5 illustrates a directed graph used to generate an end-to-enddistributed trace, according to an example embodiment.

FIG. 6 illustrates a hardware block diagram of a computing deviceconfigured to perform functions associated with operations discussedherein, according to an example embodiment.

FIG. 7 illustrates a flowchart of a method for performing functionsassociated with operations discussed herein, according to an exampleembodiment.

DESCRIPTION OF EXAMPLE EMBODIMENTS Overview

Techniques are described herein for generating an end-to-end distributedtrace in connection with a cloud or datacenter environment. In oneexample embodiment, a server obtains target application telemetry dataand external telemetry data associated with one or more correlationidentifiers included in one or more network communications provided to atarget application in the cloud or datacenter environment. The serveraggregates the target application telemetry data and the externaltelemetry data based on the one or more correlation identifiers togenerate an end-to-end distributed trace associated with the one or morenetwork communications.

EXAMPLE EMBODIMENTS

FIG. 1 illustrates a system 100 configured for end-to-end distributedtracing, according to an example embodiment. System 100 includes agent110, front-end services 120, cloud or datacenter environment 130, andback-end services 140. Agent 110, which may be a synthetic tracingagent, includes synthetic network agent 144 and/or synthetic web agent146. Cloud or datacenter environment 130 hosts target/primaryapplication 150. System 100 also includes network nodes 160(1)-160(6)(e.g., routers, switches, etc.) configured to transmit networkcommunications between agent 110, front-end services 120, cloud ordatacenter environment 130, and back-end services 140 (e.g., over theInternet).

Agent 110 may be a remote/external, synthetic, dedicated test device.Front-end services 120 and back-end services 140 may be configured tohelp establish and maintain connections between end users (and/or agent110) and target application 150. Front-end services 120 may includeDomain Name System (DNS) services, Content Delivery Networks (CDNs),authentication services (e.g., Identity Provider (IdP) services),framework provider services, Secure Access Service Edge (SASE), adware,etc. Back-end services 140 may include cloud and Internet services suchas cloud Application Programming Interface (API) services.

In one example, cloud or datacenter environment 130 may be a Kubernetes®environment. Target application 150 may be a web or mobile application.Target application 150 may include multiplecomponents/routines/microservices that are connected within cloud ordatacenter environment 130. The microservices may include a front-endmicroservice, an ad microservice, a checkout microservice, an emailmicroservice, a payment microservice, a shipping microservice, acurrency microservice, a product catalog microservice, a recommendationmicroservice, and a cart microservice configured to utilize a cache. Inoperation, the front-end service may obtain incoming applicationrequests (e.g., Hypertext Transfer Protocol (HTTP) requests), such asweb user requests, from the load generator and/or the Internet. Forexample, the application requests may correspond to an end userpurchase, and the microservices within cloud or datacenter environment130 may perform one or more respective operations to fulfill the enduser purchase.

To measure the performance of target application 150 and thecorresponding microservices, engineers/operators associated with targetapplication 150 might employ distributed tracing techniques. However,conventional distributed tracing techniques are limited to monitoringperformance of target application 150 within cloud or datacenterenvironment 130 for a given application request. Thus, conventionaldistributed tracing techniques cannot provide performance visibility foran application request outside the cloud or datacenter stack of targetapplication 150.

In particular, conventional distributed tracing techniques havesignificant “blind spots” at agent 110, front-end services 120, andback-end services 140. These blind spots also exist at the non-trivialnetwork paths comprising network nodes 160(1)-160(6), including thenetwork paths between agent 110 and front-end services 120, agent 110and cloud or datacenter environment 130, front-end services 120 andcloud or datacenter environment 130, and cloud or datacenter environment130 and back-end services 140. These blind spots are not traceable; thatis, conventional distributed tracing techniques cannot integrate tracemetrics to cover these blind spots with trace metrics generated withincloud or datacenter environment 130. As a result, conventionaldistributed tracing techniques provide limited value to theengineers/operators, because the performance of target application 150can, from an end-user perspective, be significantly impacted by theseblind spots.

Thus, to provide end-to-end distributed tracing for a given applicationrequest, aggregation server 170 is provided with aggregation logic 180.Briefly, aggregation logic 180 may enable aggregation server 170 tostitch together data from agent 110, front-end services 120, back-endservices 140, and/or network nodes 160(1)-160(6), with data from thecloud or datacenter environment 130 to improve end-to-end applicationtracing and performance.

In one example, aggregation server 170 may obtain target applicationtelemetry data and external telemetry data associated with one or morecorrelation identifiers. The one or more correlation identifiers may beincluded in one or more network communications provided to targetapplication 150 (e.g., from agent 110). Based on the one or morecorrelation identifiers, aggregation server 170 may aggregate the targetapplication telemetry data and the external telemetry data to generatean end-to-end distributed trace associated with the one or more networkcommunications. As used herein, the term “end-to-end distributed trace”may refer to a distributed trace that incorporates both targetapplication telemetry data and external telemetry data.

The target application telemetry data may include telemetry datarelating to target application 150. The external telemetry data mayinclude telemetry data relating to entities external to targetapplication 150 and/or cloud or datacenter environment 130. For example,the external telemetry data may relate to agent 110, front-end services120, back-end services 140, and/or network nodes 160(1)-160(6). It willbe appreciated that the end-to-end distributed trace need notnecessarily incorporate external telemetry data relating to each ofagent 110, front-end services 120, back-end services 140, and/or networknodes 160(1)-160(6)—the end-to-end distributed trace may, for example,incorporate external telemetry data relating to one or more of agent110, front-end services 120, back-end services 140, and/or network nodes160(1)-160(6).

In one example, the external telemetry data may include network pathtelemetry data and/or external service telemetry data. The network pathtelemetry data may include telemetry data associated with the networkpath between agent 110 and cloud or datacenter environment 130. Thenetwork path telemetry data may be obtained from network nodes 160(2)and/or 160(3) (e.g., hop-by-hop data). The external service telemetrydata may include telemetry data associated with front-end services 120,back-end services 140, and/or the network paths between agent 110 andfront-end services 120, between front-end services 120 and cloud ordatacenter environment 130, and/or between cloud or datacenterenvironment 130 and back-end services 140. The external servicetelemetry data may be obtained from front-end services 120, back-endservices 140, and/or network nodes 160(1), 160(4), 160(5), and/or 160(6)(e.g., hop-by-hop data). In a further example, the external telemetrydata may include service chain/service function telemetry data.

The end-to-end distributed trace may start with an external requestoriginating from agent 110, rather than when the request reaches targetapplication 150. The end-to-end distributed trace may also indicate thefull end-to-end timing of application workloads from external locations(e.g., front-end services 120 and back-end services 140) and eachrelevant network hop (e.g., network nodes 160(1)-160(6)). Thus,aggregation server 170 may aggregate the portion of the application andnetwork performance between agent 110 and the front door of targetapplication 150, as well as other telemetry data, with the targetapplication telemetry data to create an end-to-end view of applicationand network performance in Full-Stack Observability (FSO) tools.

For each application request sent to target application 150, theperformance of the microservices that are involved in responding to theapplication request may be monitored. The resulting end-to-enddistributed trace may provide Site Reliability Engineering (SRE)entities, cloud operators, engineers, and Information Technology(IT)/application teams with end-to-end distributed integrated tracingmetrics that cover the full scope of, and offer granular visibilityinto, system 100. This may ultimately improve manual and/or automatedtroubleshooting and optimization of target application 150.

In one example, based on the end-to-end distributed trace, aggregationserver 170 may automatically implement a network configuration change incloud or datacenter environment 130. For example, if a given service infront-end services 120 or back-end services 140 has a long latency,aggregation server 170 may automatically cause an alternative servicewith a shorter latency to be used instead. Or, if a given network pathhas a long latency, aggregation server 170 may automatically cause analternative network path with a shorter latency to be used instead.Other embodiments may be envisioned.

With continuing reference to FIG. 1 , FIG. 2 illustrates a graphicaldepiction 200 of how components of system 100 can contribute to anend-to-end distributed trace, according to an example embodiment.Graphical depiction 200 illustrates agent 110, front-end services 120,cloud or datacenter environment 130, and back-end services 140.Graphical depiction 200 further shows network path telemetry data 210(1)obtained from network nodes 160(2) and 160(3); target applicationtelemetry data 210(2) obtained from microservices in target application150; and external service telemetry data 210(3) obtained from networknodes 160(5) and 160(6). Network path telemetry data 210(1), targetapplication telemetry data 210(2), and external service telemetry data210(3) (collectively, “telemetry data 210(1)-210(3)”), may reflect oneor more operations performed by network nodes 160(2) and 160(3), themicroservices, and network nodes 160(5) and 160(6). Each of the one ormore operations may have an associated timestamp.

Upon obtaining telemetry data 210(1)-210(3), aggregation server 170 maydetermine that telemetry data 210(1)-210(3) are correlated (e.g., basedon one or more correlation identifiers associated with telemetry data210(1)-210(3)), and aggregate telemetry data 210(1)-210(3) intoend-to-end distributed trace 220. For example, aggregation server 170may arrange the one or more operations by timestamp. It will beappreciated that any suitable combination of telemetry data may beaggregated into end-to-end distributed trace 220. For instance, in otherexamples, front-end services 120 and/or back-end services 140 mayprovide, to aggregation server 170, further external service telemetrydata which may be aggregated into end-to-end distributed trace 220.

It will be appreciated that the techniques described herein may becompatible with any suitable mechanism for collecting telemetry data.One such mechanism may be the OpenTelemetry tracing specification. TheOpenTelemetry tracing specification specifies a collection of tools,APIs, and Software Development Kits (SDKs) that are useful to generate,collect, and export telemetry data (metrics, logs, and traces) toanalyze software performance. With OpenTelemetry technology, distributedtracing may involve measuring the performance and behavior of complex,distributed software systems using the concepts of spans and traces. Aspan may represent an individual unit of work (e.g., one or moreoperations) performed by a microservice involved in fulfilling anapplication request. A trace may include one or more spans, and mayrepresent the processing of an application request by an application andcorresponding microservices required to fulfill the request. Spans andtraces may help measure the duration of the corresponding units of workand, therefore, the duration of application request processing withinthe application. OpenTelemetry data may be provided by entities referredto as resources.

The techniques described herein may use a tracing tool such as theOpenTelemetry tracing specification to automatically generate bothinternal application traces as well as external network performance(e.g., external network paths) and external network service traces(e.g., DNS traces), and aggregate the resulting external networkperformance and external service data with the resultingapplication-oriented distributed tracing data. Thus, using a syntheticweb agent and a synthetic network agent, end-to-end distributedapplication traces may be created which broaden the scope andeffectiveness of existing observability tools such as the OpenTelemetrytracing specification.

With continuing reference to FIG. 1 , FIG. 3 illustrates a system 300configured for end-to-end distributed tracing using techniques specifiedby the OpenTelemetry tracing specification, according to an exampleembodiment. System 300 includes agent 110 (including synthetic networkagent 144 and/or synthetic web agent 146), target application 150,aggregation server 170, and span processor controller 310. Targetapplication 150 and span processor controller 310 include OpenTelemetrycollectors 320(1) and 320(2), respectively. OpenTelemetry collectors320(1) and 320(2) may be configured to generate distributed tracingdata.

In this example, target application 150 has a Uniform Resource Locator(URL) of “my.app.com.” Agent 110 may be a ThousandEyes® agent, and spanprocessor controller 310 may be a ThousandEyes collector. TheThousandEyes software monitoring platform may provide visibility andenable actions to maintain and optimize digital services, such asapplication experience, hop-by-hop network path, and performance. TheThousandEyes software monitoring platform provides visibility from anend-user to an application, over a network. Furthermore, aggregationserver 170 may be an OpenTelemetry back-end configured to aggregateOpenTelemetry data and display the aggregated data using a visualizationtool such as the AppDynamics® application performance managementplatform and solution.

At operation 330, agent 110 provides, to target application 150, one ormore network communications. The one or more network communicationscomprise synthetic request 335 and network probe 340. Synthetic request335 may be a browser-based, application-layer, HTTP web request (e.g.,an HTTP GET) configured to initiate an HTTP transaction on targetapplication 150. In one example, synthetic request 335 may be generatedand sent by synthetic web agent 146, and network probe 340 may begenerated and sent by synthetic network agent 144.

To provide synthetic request 335, synthetic web agent 146 may create anew distributed trace span object associated with synthetic request 335.The span object may include span context (e.g., cross-layer context)including a unique span identifier that can be used to uniquely identifythe span object (e.g., a root span). The span object may also includetrace context including correlation identifier 345. In one example,correlation identifier 345 may be a unique trace identifier of anend-to-end distributed trace associated with the one or more networkcommunications (e.g., an OpenTelemetry trace identifier configured toidentify the distributed trace root span). The unique trace identifiermay be created because the root span, initiated by synthetic web agent146, is the first span in (e.g., represents the start of) the end-to-enddistributed trace. Any subsequent spans in the end-to-end distributedtrace may be child spans of this root span, each child span associatedwith a unique span identifier.

The span object may also include other context regarding syntheticrequest 335 as custom OpenTelemetry properties. The custom OpenTelemetryproperties may include metadata of agent 110, such as an identifier(e.g., name) of agent 110, a location (e.g., a geolocation, such ascity, state, country, metro region, etc. and/or a geotag, such aslatitude/longitude) of agent 110, a service provider (e.g., networkprovider or Internet Service Provider (ISP)) of agent 110, host/providernetwork of agent 110, etc. The custom OpenTelemetry properties mayfurther include a network host, the Internet Protocol (IP) address oftarget application 150, the URL of target application 150(“my.app.com”), external service names and/or URLs, the identifier ofthe synthetic test (to allow linking back to synthetic test), and/orparameters regarding the method of transporting the telemetry data.These custom OpenTelemetry properties may be used to enrich theresulting end-to-end distributed trace.

Synthetic web agent 146 may inject correlation identifier 345 (e.g., theunique trace identifier), the unique span identifier of the root span,and/or the other context into synthetic request 335. Correlationidentifier 345 and/or the unique span identifier may be injectedin-band, using an HTTP header format that is recognizable by targetapplication 150. In one example, correlation identifier 345 and/or theunique span identifier may be injected into a World Wide Web Consortium(W3C) trace context HTTP header, which may be compatible withdistributed tracing frameworks like techniques specified by theOpenTelemetry tracing specification. In another example, the correlationidentifier 345 and/or the unique span identifier may be embedded intothe payload of synthetic request 335.

Synthetic request 335 may simulate an end user action on a browser, suchas selecting (e.g., clicking on) a few products and placing an order.But, instead of simply mimicking what an end user would do, thesynthetic test initiated by synthetic web agent 146 may also involveinjecting correlation identifier 345 and/or other trace data to enableend-to-end distributed tracing.

Synthetic network agent 144 may send network probe 340 to the InternetProtocol (IP) address and/or hostname of target application 150. Networkprobe 340 may be sent out-of-band with respect to synthetic request 335.Synthetic network agent 144 may generate and/or provide network probe340 to target application 150 before, while, or after synthetic webagent 146 generates and/or provides synthetic request 335 to targetapplication 150. In one specific example, agent 110 may embedcorrelation identifier 345 in synthetic request 335 while simultaneouslygenerating network probe 340. In another specific example, agent 110 maygenerate network probe 340 while target application 150 is processingsynthetic request 335.

Synthetic network agent 144 may tag network probe 340 with an identifierthat enables correlation with synthetic request 335. In one example,synthetic network agent 144 may tag or inject network probe 340 withcorrelation identifier 345. In another example, synthetic network agent144 may tag network probe 340 with another correlation identifier thatis associated with correlation identifier 345. In this example,correlation identifier 345 may be associated with target applicationtelemetry data that is generated responsive to synthetic request 335,and the other correlation identifier may be associated with externaltelemetry data that is generated responsive to one or more networkprobes (e.g., network probe 340). In still another example, correlationidentifier 345 or the other correlation identifier may betagged/assigned to the network telemetry data collected by agent 110,responsive to network probe 340, for later correlation with the targetapplication telemetry data. Correlation identifier 345 and the othercorrelation identifier may be associated with each other to enablecorrelation of the target application telemetry data and the externaltelemetry data in an end-to-end distributed trace.

Synthetic request 335 and network probe 340 may act as distributedtracing resources, e.g., participants in networking operations.Synthetic request 335 and network probe 340 may generate telemetry datadepending on the performance metrics that synthetic request 335 andnetwork probe 340 are configured to measure. Examples of generatedtelemetry data may include latency, jitter, packet loss, DNS responsetime, Secure Sockets Layer (SSL) connection time, server response time,etc.

Synthetic request 335 may be configured to generate target applicationtelemetry data, and network probe 340 may be configured to generateexternal telemetry data. In this example, network probe 340 may beconfigured for network performance and path discovery, and may thereforegenerate network path telemetry data. The network path telemetry datamay include hop-by-hop data detailing the network path traveled bysynthetic request 335 from agent 110 to target application 150. Ifsynthetic request 335 and network probe 340 are provided in parallel(e.g., at the same time or nearly the same time), the network pathtelemetry data may represent network performance at the time (or at theapproximate time) that synthetic request 335 was made.

Synthetic network agent 144 and/or target application 150 may alsoprovide additional network probes. In one example, synthetic networkagent 144 and/or target application 150 may provide additional networkprobes to external services such as front-end services 120 and/orback-end services 140; those additional network probes may generateexternal service telemetry data. In another example, synthetic networkagent 144 and/or target application 150 may provide additional networkprobes to service functions and/or service chains; those additionalnetwork probes may generate service function/chain telemetry data.

At operation 350, target application 150 obtains synthetic request 335,and in response, generates target application telemetry data for theend-to-end distributed trace. In generating the target applicationtelemetry data, target application 150 may handle/process syntheticrequest 335 similar to a request obtained from an end user, e.g., byprompting various microservices to perform one or more operations tofulfill synthetic request 335. Target application 150 may also inspectsynthetic request 335 for a header (e.g., a traceparent header) thatincludes correlation identifier 345 and/or the span identifier.

Target application 150 may extract correlation identifier 345 fromsynthetic request 335 for association with any child spans generated bytarget application 150 as a result of processing synthetic request 335.Target application 150 may also extract the span identifier and set thespan identifier as the parent span identifier for the end-to-enddistributed trace. Thus, target application 150 may mark any child spansgenerated by target application 150 as a result of processing syntheticrequest 335, as child spans of the span referenced by synthetic request335 (e.g., the span identifier).

Target application 150 may also extract other context injected intosynthetic request 335, such as the identifier of agent 110, the locationof agent 110, the service provider of agent 110, the IP address oftarget application 150, the URL of target application 150, etc. Targetapplication 150 may forward the other context to any subsequentdistributed tracing calls that will become part of the end-to-enddistributed trace. In one example, target application 150 may use theOpenTelemetry open-source distributed tracing framework to extractcontext from synthetic request 335 and apply the context to subsequentspans generated by tracing within target application 150.

In one example, target application 150 may include a visualization agent(e.g., an AppDynamics agent) configured to extract correlationidentifier 345 (and/or other context) from synthetic request 335. Thevisualization agent may generate OpenTelemetry spans and mark thosespans as child spans of the trace identified in synthetic request 335.The visualization agent may mark those spans as child spans, rather thanroot spans, based on correlation identifier (e.g., the trace identifier)in synthetic request 335. The visualization agent may also ingest othercontext (e.g., trace context), from synthetic request 335. Thevisualization agent may automatically extract context from syntheticrequest 335, for example, when the context is present in the traceparentheader.

At operation 355, aggregation server 170 obtains the target applicationtelemetry data from target application 150 responsive to syntheticrequest 335. Aggregation server 170 may obtain the target applicationtelemetry data from OpenTelemetry collector 320(1) or directly fromtarget application 150. The target application telemetry data mayinclude OpenTelemetry spans accumulated/generated from any instrumenteddistributed microservices within target application 150 that are used toprocess synthetic request 335. The OpenTelemetry spans may include childspans of the initial root span created by agent 110. The targetapplication telemetry data may include correlation identifier 345.

At operation 360, agent 110 gathers external telemetry data (e.g.,metrics obtained from one or more external services and/or networkpath/hop-by-hop data), generates one or more corresponding spans, andassigns to each span a unique span identifier. Agent 110 provides theexternal telemetry data (e.g., raw network probe data) to span processorcontroller 310 to be provided to aggregation server 170 as a singletransaction. The external telemetry data may include correlationidentifier 345. Agent 110 may also provide additional context, such astimestamps, location of agent 110, the service provider of agent 110,etc.

At operation 365, OpenTelemetry collector 320(2) may ingest nativetelemetry test data (e.g., external telemetry data) from agent 110,process the external telemetry data, and decompose (break down) theexternal telemetry data into multiple, individual OpenTelemetry tracingspans representing each hop and relevant external service. In oneexample, OpenTelemetry collector 320(2) may be running on aSoftware-as-a-Service (SaaS) that includes span processor controller310.

OpenTelemetry collector 320(2) may intelligently create theOpenTelemetry spans from the extracted hop-by-hop path data and externalnetwork/HTTP service queries (e.g., DNS resolution, connect, SSL,authenticate, etc.). OpenTelemetry collector 320(2) may assign, to eachspan, a parent span identifier corresponding to the span identifier ofthe initial root span created by agent 110. As a result, theOpenTelemetry spans may be child spans of the single, root span fortop-level metrics. Thus, both the child spans generated by OpenTelemetrycollector 320(1) and the child spans generated by OpenTelemetrycollector 320(2) may be associated with the root span identifier (andcorrelation identifier 345).

Alternatively, in the absence of span processor controller 310,OpenTelemetry collector 320(2) may be running on agent 110, and agent110 may process the external telemetry data (e.g., network probe data)itself. In this example, agent 110 may break down the external telemetrydata into multiple OpenTelemetry spans (e.g., one for one for each hopand a root span for top level metrics), and associate the OpenTelemetryspans with the root span identifier (and correlation identifier 345).

At operation 370, span processor controller 310 emits the OpenTelemetryspans to aggregation server 170. Alternatively, in the absence of spanprocessor controller 310, agent 110 may generate and send theOpenTelemetry spans directly to aggregation server 170. In any event,aggregation server 170 may directly or indirectly obtain (e.g., collect)the external telemetry data from agent 110.

At operation 375, aggregation server 170 aggregates the distributedtracing span data to generate an end-to-end distributed trace associatedwith synthetic request 335. The distributed tracing span data mayinclude the initial span generated by synthetic web agent 146 thatcorresponds to synthetic request 335, target application telemetry dataobtained from OpenTelemetry collector 320(1), network path telemetrydata obtained from OpenTelemetry collector 320(2), and external servicetelemetry data obtained from OpenTelemetry collector 320(1) and/orOpenTelemetry collector 320(2).

Aggregation server 170 may aggregate the target application telemetrydata, the network path telemetry data based on correlation identifier345. In one example, aggregation server 170 may perform the aggregationin response to obtaining the target application telemetry data withcorrelation identifier 345 and the network path telemetry data withcorrelation identifier 345. In another example, aggregation server 170may perform the aggregation in response to obtaining the targetapplication telemetry data with correlation identifier 345 and thenetwork path telemetry data with another correlation identifier that isassociated with correlation identifier 345. Because the OpenTelemetryspans are all correlated (directly or indirectly) to the original rootspan, aggregation server 170 may assemble, from these spans, theend-to-end distributed trace for synthetic request 335.

In addition to correlation identifier 345 (e.g., the unique traceidentifier), a unique span identifier, and the root span identifier, oneor more spans may also include additional context, such as metadata ofagent 110 and/or other context. For example, one or more spans mayinclude the identifier, location, service provider, and/or host/providernetwork of agent 110, and/or the network host, IP address and/or URL oftarget application 150, external service names and/or URLs, parametersregarding the method of transporting the telemetry data, etc.

Aggregation server 170 may associate this additional context with theend-to-end distributed trace. For example, aggregation server 170 mayassociate metadata of agent 110 (e.g., a location or a network serviceprovider of agent 110) with the end-to-end distributed trace.Aggregation server 170 may configure the spans via API with theappropriate context (e.g., “correlationId,” “applicationId,”“transactionId,” “service,” etc.) to aggregate the context with theexternal telemetry dataset and the target application telemetry dataset(e.g., internal tracing data). Thus, aggregation server 170 may enrichthe end-to-end distributed trace with the additional context such as themetadata of agent 110.

Aggregation server 170 may craft individual trace spans from hop-by-hoppath data by filtering out various anomalies. For example, aggregationserver 170 may parse out requests to external services extracted fromHTTP request sessions (e.g., DNS lookup) and/or browser waterfall files(e.g., CDN queries, authentication, etc.).

In one example, operations 330, 350, 355, 360, 365, 370, and 375 may berepeated for any number of geographically dispersed/remote agents. Forinstance, multiple agents may each send a synthetic request to targetapplication 150 and generate one or more network probes. Ultimately,aggregation server 170 may produce multiple end-to-end distributiontraces, each associated with the location of a respective agent.

In another example, operations 330, 350, 355, 360, 365, 370, and 375 maybe repeated over time (e.g., agent 110 may send multiple syntheticrequests, etc.). Aggregation server 170 may collect correspondingOpenTelemetry spans and build multiple end-to-end distributed traces foragent 110, each corresponding to a different time.

Aggregation server 170 may display the end-to-end distributed trace toenable the owner of target application 150 to view the end-to-enddistributed trace(s). For instance, the owner of target application 150may view end-to-end distributed traces from global vantage points. Theowner may parse the collection of end-to-end distributed traces based onthe synthetic requests (e.g., filtering out other production/usertraffic) and compare performance for different geographic regions. Thedisplay may also/alternatively enable the owner to observe how the tracetopology changes over time for a given agent (e.g., agent 110).

FIG. 4 illustrates a display 400 of an end-to-end distributed trace,according to an example embodiment. Display 400 may be generated byaggregation server 170. Display 400 includes overview window 410 andbreakdown window 420. Overview window 410 indicates a timestamp at whichthe end-to-end distributed trace started (e.g., when an agent sent asynthetic request to a target application), a duration of the end-to-enddistributed trace, and a total number of spans present in the end-to-enddistributed trace, among other data. Overview window 410 also includes agraph displaying the entire end-to-end distributed trace.

Breakdown window 420 includes an aggregation of the spans that make upthe end-to-end distributed trace, arranged in a chronological view. Thespans may represent monitored performance/responses of the correspondingmicroservices shown under the “Service & Operation” tab. As shown, theend-to-end distributed trace may be broken down into Internet andnetwork hops, and may include individual spans for each network hop inthe path. The top span corresponding to the “cart” microservice mayrepresent the root span; the remaining spans underneath the root spanmay be child spans of the root span. The end-to-end distributed tracemay include the root span and the child spans. Each span in theend-to-end distributed trace may have a unique span identifier.

In this example, the end-to-end distributed trace includes network pathtelemetry data, corresponding to Hops 1-5, and network service (e.g.,service chain/function) telemetry data, corresponding to a firewall anda load balancer. The end-to-end distributed trace also includes targetapplication telemetry data, corresponding to frontend and associatedmicroservices. Because the spans were generated using at least onecorrelation identifier (e.g., a trace identifier), the network pathtelemetry data and the target application telemetry data may be stitchedtogether in a single, end-to-end distributed trace.

This “stitching” may allow engineers/operators to consume themicroservices involved in responding to a given synthetic request andidentify which microservice(s) might be performing poorly. For example,the end-to-end distributed trace shown in display 400 indicates that Hop5 experienced high latency. Accordingly, an alternative network path maybe manually or automatically chosen to avoid Hop 5.

FIG. 5 illustrates a directed graph 500 used to generate an end-to-enddistributed trace, according to an example embodiment. Directed graph500 includes nodes 510(1)-510(7) and edges 520(1)-520(8). Nodes 510(1)and 510(2) represent agents located in Seattle, WA and San Francisco,CA, respectively, and nodes 510(3)-510(7) represent the frontend,currencyservice, checkoutservice, paymentservice, andproductcatalogueservice microservices, respectively. Edges 520(1)-520(8)represent one or more network communications between nodes510(1)-510(7). For example, edges 520(1) and 520(2) may representrespective synthetic requests, and edges 520(3)-520(8) may representoperations triggered responsive to the synthetic requests. Each edge520(1)-520(8) is associated with a number, representing the number ofoperations that occur between connected pairs of nodes 510(1)-510(7). Anend-to-end distributed trace may be generated based on the informationin directed graph 500, such as the agents, their locations, and the oneor more network communications and operations represented by edges520(1)-520(8).

Referring to FIG. 6 , FIG. 6 illustrates a hardware block diagram of acomputing device 600 that may perform functions associated withoperations discussed herein in connection with the techniques depictedin FIGS. 1-5 . In various embodiments, a computing device, such ascomputing device 600 or any combination of computing devices 600, may beconfigured as any entity/entities as discussed for the techniquesdepicted in connection with FIGS. 1-5 in order to perform operations ofthe various techniques discussed herein.

In at least one embodiment, computing device 600 may include one or moreprocessor(s) 602, one or more memory element(s) 604, storage 606, a bus608, one or more network processor unit(s) 610 interconnected with oneor more network input/output (I/O) interface(s) 612, one or more I/Ointerface(s) 614, and control logic 620. In various embodiments,instructions associated with logic for computing device 600 can overlapin any manner and are not limited to the specific allocation ofinstructions and/or operations described herein.

In at least one embodiment, processor(s) 602 is/are at least onehardware processor configured to execute various tasks, operationsand/or functions for computing device 600 as described herein accordingto software and/or instructions configured for computing device 600.Processor(s) 602 (e.g., a hardware processor) can execute any type ofinstructions associated with data to achieve the operations detailedherein. In one example, processor(s) 602 can transform an element or anarticle (e.g., data, information) from one state or thing to anotherstate or thing. Any of potential processing elements, microprocessors,digital signal processor, baseband signal processor, modem, PHY,controllers, systems, managers, logic, and/or machines described hereincan be construed as being encompassed within the broad term ‘processor’.

In at least one embodiment, memory element(s) 604 and/or storage 606is/are configured to store data, information, software, and/orinstructions associated with computing device 600, and/or logicconfigured for memory element(s) 604 and/or storage 606. For example,any logic described herein (e.g., control logic 620) can, in variousembodiments, be stored for computing device 600 using any combination ofmemory element(s) 604 and/or storage 606. Note that in some embodiments,storage 606 can be consolidated with memory elements 604 (or viceversa), or can overlap/exist in any other suitable manner.

In at least one embodiment, bus 608 can be configured as an interfacethat enables one or more elements of computing device 600 to communicatein order to exchange information and/or data. Bus 608 can be implementedwith any architecture designed for passing control, data and/orinformation between processors, memory elements/storage, peripheraldevices, and/or any other hardware and/or software components that maybe configured for computing device 600. In at least one embodiment, bus608 may be implemented as a fast kernel-hosted interconnect, potentiallyusing shared memory between processes (e.g., logic), which can enableefficient communication paths between the processes.

In various embodiments, network processor unit(s) 610 may enablecommunication between computing device 600 and other systems, entities,etc., via network I/O interface(s) 612 to facilitate operationsdiscussed for various embodiments described herein. In variousembodiments, network processor unit(s) 610 can be configured as acombination of hardware and/or software, such as one or more Ethernetdriver(s) and/or controller(s) or interface cards, Fibre Channel (e.g.,optical) driver(s) and/or controller(s), and/or other similar networkinterface driver(s) and/or controller(s) now known or hereafterdeveloped to enable communications between computing device 600 andother systems, entities, etc. to facilitate operations for variousembodiments described herein. In various embodiments, network I/Ointerface(s) 612 can be configured as one or more Ethernet port(s),Fibre Channel ports, and/or any other I/O port(s) now known or hereafterdeveloped. Thus, the network processor unit(s) 610 and/or network I/Ointerfaces 612 may include suitable interfaces for receiving,transmitting, and/or otherwise communicating data and/or information ina network environment.

I/O interface(s) 614 allow for input and output of data and/orinformation with other entities that may be connected to computingdevice 600. For example, I/O interface(s) 614 may provide a connectionto external devices such as a keyboard, keypad, a touch screen, and/orany other suitable input device now known or hereafter developed. Insome instances, external devices can also include portable computerreadable (non-transitory) storage media such as database systems, thumbdrives, portable optical or magnetic disks, and memory cards. In stillsome instances, external devices can be a mechanism to display data to auser, such as, for example, a computer monitor, a display screen, or thelike.

In various embodiments, control logic 620 can include instructions that,when executed, cause processor(s) 602 to perform operations, which caninclude, but not be limited to, providing overall control operations ofcomputing device 600; interacting with other entities, systems, etc.described herein; maintaining and/or interacting with stored data,information, parameters, etc. (e.g., memory element(s), storage, datastructures, databases, tables, etc.); combinations thereof; and/or thelike to facilitate various operations for embodiments described herein.

The programs described herein (e.g., control logic 620) may beidentified based upon application(s) for which they are implemented in aspecific embodiment. However, it should be appreciated that anyparticular program nomenclature herein is used merely for convenience;thus, embodiments herein should not be limited to use(s) solelydescribed in any specific application(s) identified and/or implied bysuch nomenclature.

In various embodiments, entities as described herein may storedata/information in any suitable volatile and/or non-volatile memoryitem (e.g., magnetic hard disk drive, solid state hard drive,semiconductor storage device, Random Access Memory (RAM), Read OnlyMemory (ROM), Erasable Programmable ROM (EPROM), Application SpecificIntegrated Circuit (ASIC), etc.), software, logic (fixed logic, hardwarelogic, programmable logic, analog logic, digital logic), hardware,and/or in any other suitable component, device, element, and/or objectas may be appropriate. Any of the memory items discussed herein shouldbe construed as being encompassed within the broad term ‘memoryelement’. Data/information being tracked and/or sent to one or moreentities as discussed herein could be provided in any database, table,register, list, cache, storage, and/or storage structure: all of whichcan be referenced at any suitable timeframe. Any such storage optionsmay also be included within the broad term ‘memory element’ as usedherein.

Note that in certain example implementations, operations as set forthherein may be implemented by logic encoded in one or more tangible mediathat is capable of storing instructions and/or digital information andmay be inclusive of non-transitory tangible media and/or non-transitorycomputer readable storage media (e.g., embedded logic provided in: anASIC, Digital Signal Processing (DSP) instructions, software[potentially inclusive of object code and source code], etc.) forexecution by one or more processor(s), and/or other similar machine,etc. Generally, memory element(s) 604 and/or storage 606 can store data,software, code, instructions (e.g., processor instructions), logic,parameters, combinations thereof, and/or the like used for operationsdescribed herein. This includes memory elements 604 and/or storage 606being able to store data, software, code, instructions (e.g., processorinstructions), logic, parameters, combinations thereof, or the like thatare executed to carry out operations in accordance with teachings of thepresent disclosure.

In some instances, software of the present embodiments may be availablevia a non-transitory computer useable medium (e.g., magnetic or opticalmediums, magneto-optic mediums, Compact Disc ROM (CD-ROM), DigitalVersatile Disc (DVD), memory devices, etc.) of a stationary or portableprogram product apparatus, downloadable file(s), file wrapper(s),object(s), package(s), container(s), and/or the like. In some instances,non-transitory computer readable storage media may also be removable.For example, a removable hard drive may be used for memory/storage insome implementations. Other examples may include optical and magneticdisks, thumb drives, and smart cards that can be inserted and/orotherwise connected to computing device 600 for transfer onto anothercomputer readable storage medium.

FIG. 7 is a flowchart of an example method 700 performed in connectionwith a cloud or datacenter environment. Method 700 may be performed byany suitable entity, such as aggregation server 170 (FIG. 1 ). Atoperation 710, aggregation server 170 obtains target applicationtelemetry data and external telemetry data associated with one or morecorrelation identifiers included in one or more network communicationsprovided to a target application in the cloud or datacenter environment.At operation 720, aggregation server 170 aggregates the targetapplication telemetry data and the external telemetry data based on theone or more correlation identifiers to generate an end-to-enddistributed trace associated with the one or more networkcommunications.

In one example, agents that simultaneously generate syntheticapplication and network probes from diverse geographic regions may berepresented as resources that originate the generation of a distributedapplication trace. Additional application instrumentation tooling neednot necessarily be added to client-side code to obtain the trace.Representing external synthetic agents as resources within a distributedtrace may permit injection, into the trace/span, of context attributesof the agent (e.g., geo-location of agent/request, originating networkservice provider, number of network hops, type of network interface,links back to synthetic test context, etc.).

These context attributes may enrich the overall distributed trace inmeaningful way, and may be otherwise unavailable an end-to-enddistributed trace. This enables the application owner to make end-to-endobservability queries such as, “display the traces originating from theSeattle metro region to ‘myapp.com’ for which Comcast is the ISP,” or“display the end-to-end trace of requests to ‘myapp.com’ originatingfrom the Chicago metro region across AT&T networks for which thepurchase item business transaction exceeds normal thresholds.”

As described herein, target application trace data may be aggregatedwith network trace data into a common distributed trace. This mayinvolve breaking down multi-hop network path data and network servicerequest data (e.g., DNS, connect, SSL, etc.) into distributed tracingspans that can be consumed by a framework like techniques specified bythe OpenTelemetry tracing specification.

The synthetic test web (HTTP) request may be used as the vehicle forcreating an association between the network trace context and the targetapplication trace context. This approach to correlating the networkcontext and application using HTTP injection “in-band” may obviate theneed for a brute force approach (e.g., using a centralized service andAPI calls to create the correlation out-of-band).

The techniques described herein may enable integration/aggregation ofexternal services, network performance data, and internal applicationtraces with distributed tracing spans, thereby creating end-to-enddistributed application traces. This may be accomplished using syntheticagents running in both external geographic locations and within a targetapplication environment. The agents may simultaneously generatesynthetic application test data, network path data, and other data usingnetwork communications sent to the target application and/or externalbackend services. The generated data may be converted to OpenTelemetrytrace data to enable correlation of the external traces with internalapplication traces and thereby produce a single end-to-end trace. Theend-to-end trace may be displayed/visualized in an OpenTelemetry backendtool.

The agent(s) may be represented as distributed tracing resources (e.g.,OpenTelemetry resources) and provide meaningful details about themselvesusing OpenTelemetry resource attributes and semantics. As a result,these details may be properly identified by an OpenTelemetry backend. Inone example, an end-to-end synthetic network operation may berepresented by an aggregation of the basic network metrics (latency,loss, and jitter) generated by one or more network probes and devicesinvolved in carrying the communication between two endpoints.

Synthetic network operations may be represented with OpenTelemetrytraces, spans, and attributes. In one example, each synthetic networkoperation may be executed within the context of an OpenTelemetry trace;each probe or network device participating in the operation may berepresented as an OpenTelemetry resource (e.g., source of telemetrydata) within the trace; each segment of communication between probes anddevices may be represented as an OpenTelemetry span; and the jitter(network.jitter) and loss (network.loss) metrics generated for eachsegment may be added as attributes to their respective OpenTelemetryspans, while the latency metric may be represented by both theduration/response time of the span as attributes network.latencyAvg,network.latencyMin, network.latencyMax, etc.

Techniques described herein may leverage capabilities from a syntheticapplication and network probing agent, (e.g., a ThousandEyes agent),distributed tracing framework (e.g., techniques specified by theOpenTelemetry tracing specification), and distributed tracingbackend/visualization tools (e.g., an AppDynamics applicationperformance management platform and solution). In one example, aThousandEyes synthetic test agent may be used to generate external pathand application trace data, and the AppDynamics application performancemanagement platform and solution may be used to instrument applicationsand as an OpenTelemetry observability platform to view aggregatedOpenTelemetry trace data. ThousandEyes controllers/agents may beenhanced to generate OpenTelemetry span data from synthetic test data;or, alternatively, a custom OpenTelemetry collector may be used toconvert ThousandEyes agent telemetry into OpenTelemetry spans.Furthermore, application instrumentation may be enhanced to detectOpenTelemetry trace context in incoming web requests.

In accordance with techniques described herein, external service andnetwork performance data may be captured and consumed by distributedtracing frameworks (e.g., techniques specified by the OpenTelemetrytracing specification). That data may be aggregated with typicalcloud-native distributed tracing data. Synthetic test data may begenerated simultaneously with path data from external locations.Synthetic testing agents may run as individual OpenTelemetry collectorsconfigured to collect application and network metrics that may beconsumed by an observability tooling/system that is based on theOpenTelemetry tracing specification. Thus, synthetic application testingand synthetic network probing technology may be integrated withdistributed tracing tooling (e.g., the OpenTelemetry tracingspecification) in order to make the external path and serviceperformance data available.

Embodiments described herein may include one or more networks, which canrepresent a series of points and/or network elements of interconnectedcommunication paths for receiving and/or transmitting messages (e.g.,packets of information) that propagate through the one or more networks.These network elements offer communicative interfaces that facilitatecommunications between the network elements. A network can include anynumber of hardware and/or software elements coupled to (and incommunication with) each other through a communication medium. Suchnetworks can include, but are not limited to, any Local Area Network(LAN), Virtual LAN (VLAN), Wide Area Network (WAN) (e.g., the Internet),Software Defined WAN (SD-WAN), Wireless Local Area (WLA) access network,Wireless Wide Area (WWA) access network, Metropolitan Area Network(MAN), Intranet, Extranet, Virtual Private Network (VPN), Low PowerNetwork (LPN), Low Power Wide Area Network (LPWAN), Machine to Machine(M2M) network, Internet of Things (IoT) network, Ethernetnetwork/switching system, any other appropriate architecture and/orsystem that facilitates communications in a network environment, and/orany suitable combination thereof.

Networks through which communications propagate can use any suitabletechnologies for communications including wireless communications (e.g.,4G/5G/nG, IEEE 802.11 (e.g., Wi-Fi®/Wi-Fi6®), IEEE 802.16 (e.g.,Worldwide Interoperability for Microwave Access (WiMAX)),Radio-Frequency Identification (RFID), Near Field Communication (NFC),Bluetooth™, mm.wave, Ultra-Wideband (UWB), etc.), and/or wiredcommunications (e.g., T1 lines, T3 lines, digital subscriber lines(DSL), Ethernet, Fibre Channel, etc.). Generally, any suitable means ofcommunications may be used such as electric, sound, light, infrared,and/or radio to facilitate communications through one or more networksin accordance with embodiments herein. Communications, interactions,operations, etc. as discussed for various embodiments described hereinmay be performed among entities that may be directly or indirectlyconnected utilizing any algorithms, communication protocols, interfaces,etc. (proprietary and/or non-proprietary) that allow for the exchange ofdata and/or information.

In various example implementations, entities for various embodimentsdescribed herein can encompass network elements (which can includevirtualized network elements, functions, etc.) such as, for example,network appliances, forwarders, routers, servers, switches, gateways,bridges, load-balancers, firewalls, processors, modules, radioreceivers/transmitters, or any other suitable device, component,element, or object operable to exchange information that facilitates orotherwise helps to facilitate various operations in a networkenvironment as described for various embodiments herein. Note that withthe examples provided herein, interaction may be described in terms ofone, two, three, or four entities. However, this has been done forpurposes of clarity, simplicity and example only. The examples providedshould not limit the scope or inhibit the broad teachings of systems,networks, etc. described herein as potentially applied to a myriad ofother architectures.

Communications in a network environment can be referred to herein as‘messages’, ‘messaging’, ‘signaling’, ‘data’, ‘content’, ‘objects’,‘requests’, ‘queries’, ‘responses’, ‘replies’, etc. which may beinclusive of packets. As referred to herein and in the claims, the term‘packet’ may be used in a generic sense to include packets, frames,segments, datagrams, and/or any other generic units that may be used totransmit communications in a network environment. Generally, a packet isa formatted unit of data that can contain control or routing information(e.g., source and destination address, source and destination port,etc.) and data, which is also sometimes referred to as a ‘payload’,‘data payload’, and variations thereof. In some embodiments, control orrouting information, management information, or the like can be includedin packet fields, such as within header(s) and/or trailer(s) of packets.Internet Protocol (IP) addresses discussed herein and in the claims caninclude any IP version 4 (IPv4) and/or IP version 6 (IPv6) addresses.

To the extent that embodiments presented herein relate to the storage ofdata, the embodiments may employ any number of any conventional or otherdatabases, data stores or storage structures (e.g., files, databases,data structures, data or other repositories, etc.) to store information.

Note that in this Specification, references to various features (e.g.,elements, structures, nodes, modules, components, engines, logic, steps,operations, functions, characteristics, etc.) included in ‘oneembodiment’, ‘example embodiment’, ‘an embodiment’, ‘anotherembodiment’, ‘certain embodiments’, ‘some embodiments’, ‘variousembodiments’, ‘other embodiments’, ‘alternative embodiment’, and thelike are intended to mean that any such features are included in one ormore embodiments of the present disclosure, but may or may notnecessarily be combined in the same embodiments.

Each example embodiment disclosed herein has been included to presentone or more different features. However, all disclosed exampleembodiments are designed to work together as part of a single largersystem or method. This disclosure explicitly envisions compoundembodiments that combine multiple previously-discussed features indifferent example embodiments into a single system or method.

Note also that a module, engine, client, controller, function, logic orthe like as used herein in this Specification, can be inclusive of anexecutable file comprising instructions that can be understood andprocessed on a server, computer, processor, machine, compute node,combinations thereof, or the like and may further include librarymodules loaded during execution, object files, system files, hardwarelogic, software logic, or any other executable modules.

It is also noted that the operations and steps described with referenceto the preceding figures illustrate only some of the possible scenariosthat may be executed by one or more entities discussed herein. Some ofthese operations may be deleted or removed where appropriate, or thesesteps may be modified or changed considerably without departing from thescope of the presented concepts. In addition, the timing and sequence ofthese operations may be altered considerably and still achieve theresults taught in this disclosure. The preceding operational flows havebeen offered for purposes of example and discussion. Substantialflexibility is provided by the embodiments in that any suitablearrangements, chronologies, configurations, and timing mechanisms may beprovided without departing from the teachings of the discussed concepts.

As used herein, unless expressly stated to the contrary, use of thephrase ‘at least one of’, ‘one or more of’, ‘and/or’, variationsthereof, or the like are open-ended expressions that are bothconjunctive and disjunctive in operation for any and all possiblecombination of the associated listed items. For example, each of theexpressions ‘at least one of X, Y and Z’, ‘at least one of X, Y or Z’,‘one or more of X, Y and Z’, ‘one or more of X, Y or Z’ and ‘X, Y and/orZ’ can mean any of the following: 1) X, but not Y and not Z; 2) Y, butnot X and not Z; 3) Z, but not X and not Y; 4) X and Y, but not Z; 5) Xand Z, but not Y; 6) Y and Z, but not X; or 7) X, Y, and Z.

Additionally, unless expressly stated to the contrary, the terms‘first’, ‘second’, ‘third’, etc., are intended to distinguish theparticular nouns they modify (e.g., element, condition, node, module,activity, operation, etc.). Unless expressly stated to the contrary, theuse of these terms is not intended to indicate any type of order, rank,importance, temporal sequence, or hierarchy of the modified noun. Forexample, ‘first X’ and ‘second X’ are intended to designate two ‘X’elements that are not necessarily limited by any order, rank,importance, temporal sequence, or hierarchy of the two elements. Furtheras referred to herein, ‘at least one of’ and ‘one or more of can berepresented using the’(s)′ nomenclature (e.g., one or more element(s)).

In one form, a method performed in connection with a cloud or datacenterenvironment is provided. The method comprises: obtaining targetapplication telemetry data and external telemetry data associated withone or more correlation identifiers included in one or more networkcommunications provided to a target application in the cloud ordatacenter environment; and aggregating the target application telemetrydata and the external telemetry data based on the one or morecorrelation identifiers to generate an end-to-end distributed traceassociated with the one or more network communications.

In one example, the method further comprises: displaying the end-to-enddistributed trace.

In one example, the one or more network communications comprise asynthetic request and a network probe; and obtaining the targetapplication telemetry data and the external telemetry data includesobtaining the target application telemetry data from the targetapplication responsive to the synthetic request and obtaining theexternal telemetry data from an agent that provided the one or morenetwork communications to the target application responsive to thenetwork probe.

In one example, the one or more correlation identifiers are one or moreunique trace identifiers associated with the end-to-end distributedtrace. In a further example, the end-to-end distributed trace includesone or more spans, each span associated with a unique span identifier.

In one example, obtaining the external telemetry data includes:obtaining external service telemetry data associated with the one ormore correlation identifiers. In another example, obtaining the externaltelemetry data includes: obtaining network path telemetry dataassociated with the one or more correlation identifiers.

In one example, the method further comprises: associating metadata of anagent that provided the one or more network communications to the targetapplication, with the end-to-end distributed trace. In a furtherexample, associating the metadata of the agent with the end-to-enddistributed trace includes: associating a location of the agent with theend-to-end distributed trace. In another further example, associatingthe metadata of the agent with the end-to-end distributed traceincludes: associating a network service provider of the agent with theend-to-end distributed trace.

In one example, the method further comprises: based on the end-to-enddistributed trace, automatically implementing a network configurationchange in the cloud or datacenter environment.

In another form, an apparatus is provided. The apparatus comprises: anetwork interface configured to obtain or provide networkcommunications; and one or more processors coupled to the networkinterface, wherein the one or more processors are configured to: obtaintarget application telemetry data and external telemetry data associatedwith one or more correlation identifiers included in one or more networkcommunications provided to a target application in a cloud or datacenterenvironment; and aggregate the target application telemetry data and theexternal telemetry data based on the one or more correlation identifiersto generate an end-to-end distributed trace associated with the one ormore network communications.

In another form, one or more non-transitory computer readable storagemedia are provided. The non-transitory computer readable storage mediaare encoded with instructions that, when executed by a processor, causethe processor to: obtain target application telemetry data and externaltelemetry data associated with one or more correlation identifiersincluded in one or more network communications provided to a targetapplication in a cloud or datacenter environment; and aggregate thetarget application telemetry data and the external telemetry data basedon the one or more correlation identifiers to generate an end-to-enddistributed trace associated with the one or more networkcommunications.

One or more advantages described herein are not meant to suggest thatany one of the embodiments described herein necessarily provides all ofthe described advantages or that all the embodiments of the presentdisclosure necessarily provide any one of the described advantages.Numerous other changes, substitutions, variations, alterations, and/ormodifications may be ascertained to one skilled in the art and it isintended that the present disclosure encompass all such changes,substitutions, variations, alterations, and/or modifications as fallingwithin the scope of the appended claims.

1. A method performed in connection with a cloud or datacenterenvironment, comprising: obtaining target application telemetry data andexternal telemetry data associated with one or more correlationidentifiers included in network communications provided from an agent toa target application in the cloud or datacenter environment, wherein thenetwork communications comprise synthetic requests initiatingtransactions on the target application to generate the targetapplication telemetry data and network probes to generate the externaltelemetry data, wherein the target application telemetry data includestelemetry data related to the target application processing of thesynthetic requests in the network communications, wherein the externaltelemetry data includes telemetry data relating to entities external ofthe target application and the cloud or datacenter environment used forprocessing the synthetic requests and hop-by-hop data of network pathsfor the entities, and wherein the target application telemetry data isobtained from the target application responsive to the syntheticrequests and the external telemetry data is obtained from the agentresponsive to the network probes; aggregating the target applicationtelemetry data and the external telemetry data based on the one or morecorrelation identifiers to generate an end-to-end distributed traceassociated with the network communications; and automatically changing anetwork configuration of the cloud or datacenter environment based onthe end-to-end distributed trace.
 2. The method of claim 1, furthercomprising: displaying the end-to-end distributed trace.
 3. (canceled)4. The method of claim 1, wherein the one or more correlationidentifiers are one or more unique trace identifiers associated with theend-to-end distributed trace.
 5. The method of claim 4, wherein theend-to-end distributed trace includes one or more spans, each spanassociated with a unique span identifier.
 6. The method of claim 1,wherein obtaining the external telemetry data includes: obtainingexternal service telemetry data associated with the one or morecorrelation identifiers.
 7. The method of claim 1, wherein obtaining theexternal telemetry data includes: obtaining network path telemetry dataassociated with the one or more correlation identifiers.
 8. The methodof claim 1, further comprising: associating metadata of the agent thatprovided the network communications to the target application, with theend-to-end distributed trace.
 9. The method of claim 8, whereinassociating the metadata of the agent with the end-to-end distributedtrace includes: associating a location of the agent with the end-to-enddistributed trace.
 10. The method of claim 8, wherein associating themetadata of the agent with the end-to-end distributed trace includes:associating a network service provider of the agent with the end-to-enddistributed trace.
 11. (canceled)
 12. An apparatus comprising: a networkinterface configured to obtain or provide communications; and one ormore processors coupled to the network interface, wherein the one ormore processors are configured to: obtain target application telemetrydata and external telemetry data associated with one or more correlationidentifiers included in network communications provided from an agent toa target application in a cloud or datacenter environment, wherein thenetwork communications comprise synthetic requests initiatingtransactions on the target application to generate the targetapplication telemetry data and network probes to generate the externaltelemetry data, wherein the target application telemetry data includestelemetry data related to the target application processing of thesynthetic requests in the network communications, wherein the externaltelemetry data includes telemetry data relating to entities external ofthe target application and the cloud or datacenter environment used forprocessing the synthetic requests and hop-by-hop data of network pathsfor the entities, and wherein the target application telemetry data isobtained from the target application responsive to the syntheticrequests and the external telemetry data is obtained from the agentresponsive to the network probes; aggregate the target applicationtelemetry data and the external telemetry data based on the one or morecorrelation identifiers to generate an end-to-end distributed traceassociated with the network communications; and change a networkconfiguration of the cloud or datacenter environment based on theend-to-end distributed trace.
 13. The apparatus of claim 12, wherein theone or more processors are further configured to: display the end-to-enddistributed trace.
 14. (canceled)
 15. The apparatus of claim 12, whereinthe one or more correlation identifiers are one or more unique traceidentifiers associated with the end-to-end distributed trace.
 16. Theapparatus of claim 15, wherein the end-to-end distributed trace includesone or more spans, each span associated with a unique span identifier.17. One or more non-transitory computer readable storage media encodedwith instructions that, when executed by a processor, cause theprocessor to: obtain target application telemetry data and externaltelemetry data associated with one or more correlation identifiersincluded in network communications provided from an agent to a targetapplication in a cloud or datacenter environment, wherein the networkcommunications comprise synthetic requests initiating transactions onthe target application to generate the target application telemetry dataand network probes to generate the external telemetry data, wherein thetarget application telemetry data includes telemetry data related to thetarget application processing of the synthetic requests in the networkcommunications, wherein the external telemetry data includes telemetrydata relating to entities external of the target application and thecloud or datacenter environment used for processing the syntheticrequests and hop-by-hop data of network paths for the entities andwherein the target application telemetry data is obtained from thetarget application responsive to the synthetic requests and the externaltelemetry data is obtained from the agent responsive to the networkprobes; aggregate the target application telemetry data and the externaltelemetry data based on the one or more correlation identifiers togenerate an end-to-end distributed trace associated with the networkcommunications; and change a network configuration of the cloud ordatacenter environment based on the end-to-end distributed trace. 18.The one or more non-transitory computer readable storage media of claim17, wherein the instructions further cause the processor to: display theend-to-end distributed trace.
 19. (canceled)
 20. The one or morenon-transitory computer readable storage media of claim 17, wherein theone or more correlation identifiers are one or more unique traceidentifiers associated with the end-to-end distributed trace, andwherein the end-to-end distributed trace includes one or more spans,each span associated with a unique span identifier.
 21. The one or morenon-transitory computer readable storage media of claim 17, wherein, inobtaining the external telemetry data, the instructions cause theprocessor to: obtain external service telemetry data associated with theone or more correlation identifiers.